Multi-Sensor Real-Time Data Fusion on Embedded Computing Platforms

Continuous technological progress facilitates the development of embedded vehicle classification systems. These systems typically have a lower availability of resources than on customary personal computers. They consist of several components such as sensor data acquisition and preprocessing, feature extraction and feature-/decision-based data fusion. Besides visual, Lidarand Radar-based information acoustic data is progressively used to support the classification process. Real-time extraction and selection of appropriate features that describe the considered objects imply non-trivial problems. Features should optimally describe the object of interest in terms of maximizing the difference between the class-specific feature values and minimizing the within-class differences. Due to inherent environmental noise in real-world traffic scenarios, the features have to be robust to allow sound object representations even under varying environmental conditions such as rain, snow fall and illumination changes (day/night). In literature various acoustic features are described and used for several application domains, e.g., military applications, traffic monitoring, security surveillance and law enforcement. With few exceptions these feature extraction algorithms are implemented on standard personal computers and high-performance workstations. Thus, resource limitations and execution performance are neglected in the design and implementation of these feature extraction algorithms. Furthermore, these methods are not comprehensively evaluated concerning their computational performance on embedded platforms. The key question addressed in this thesis is how to select and implement extended techniques for acoustic feature extraction and data fusion for real-time vehicle classification on embedded platforms. The major objectives are two-fold. First, the selection and implementation of appropriate algorithms for feature extraction and data fusion. The selection criteria are based on class-specific measures that maximize the between-class variance and minimize the within-class variability of the features. Furthermore, these features satisfy predefined constraints concerning available resources and execution performance. Second, the acoustic classifiers are integrated in a multi-sensor self-training framework with the goal of significantly reducing the effort for manual labeling of training data and in an audio-visual on-line co-training framework to support autonomous learning and sceneadaption. By exploiting additional high-level fusion techniques the overall classification performance of the system is increased substantially. An embedded multi-sensor data fusion platform prototype has been developed to serve as an evaluation platform. Numerous evaluation experiments have been performed on this prototype to show the feasibility of the acoustic real-time vehicle classification approach considering embedded constraints. Moreover, the evaluation of the self-training and the audio-visual co-training framework reveal the applicability of the acoustic classifiers to support the autonomous on-line learning of visual as well as acoustic classifiers and the collaborative audio-video classification.